Augmented reality and location determination methods and apparatus

ABSTRACT

Augmented reality and location determination methods and apparatus are disclosed according to some aspects of the description. In one aspect, a location determination method includes accessing first location information regarding a location of a user interaction device in a physical world, wherein the user interaction device is configured to generate an augmented reality representation with respect to the physical world, using the first location information, identifying a marker which is proximately located with respect to the location of the user interaction device, accessing an image generated by the user interaction device which includes the marker, and processing the image to determine second location information regarding the location of the user interaction device, and wherein the second location information has increased accuracy with respect to the location of the user interaction device in the physical world compared with the first location information.

TECHNICAL FIELD

This disclosure relates to augmented reality and location determinationmethods and apparatus.

BACKGROUND

Computing systems have continually evolved and the popularity ofcomputing systems continues to increase. The advancement of computingsystems creates new uses and applications for the computing systems. Forexample, the processing speeds, storage capacities and networkcommunication speeds are constantly increasing enabling the use ofcomputing systems in increasing numbers of applications.

Furthermore, computing systems have evolved from typical office or desksystems to smaller devices, some of which have increased portability,which further expands the possible applications of the computingsystems. More specifically, notebook computers have evolved from desktopcomputers, and more recently, handheld portable devices have alsoadvanced significantly. Personal digital assistants, media players,cellular telephones, smartphones, and other portable devices haveincreased processing power and storage capacities while communicationsnetworks have also been improved allowing greater rates of data transferbetween the computing systems.

Some computing systems and networks have evolved to a sufficient extentto perform augmented reality operations which augment the physical worldwith virtual computer-generated imagery in one example. In addition,some portable computing systems have sufficient processing, storage andcommunications capabilities to provide real-time augmented reality datafor mobile users.

At least some aspects of the disclosure are directed to improvedmethods, apparatus and programming for implementing augmented realityoperations.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative representation of a user interaction deviceimplementing augmented reality operations according to one embodiment.

FIG. 2 is a functional block diagram of a media system according to oneembodiment.

FIG. 3 is a functional block diagram of a computing system according toone embodiment.

FIG. 4 is a functional block diagram of communications circuitry of auser interaction device according to one embodiment.

FIG. 5 is a functional block diagram of a user interface of a userinteraction device according to one embodiment.

FIG. 6 is a flow chart of a method implemented by a user interactiondevice to implement augmented reality operations according to oneembodiment.

FIG. 7 is a flow chart of a method implemented by a management devicewith respect to a plurality of user interaction devices according to oneembodiment.

FIG. 8 is a flow chart of a method of outputting wireless communicationssignals according to one embodiment.

FIG. 9 is a flow chart of a method of receiving wireless communicationssignals according to one embodiment.

FIGS. 10 a-10 c are illustrative representations of a method ofdetermining refined location information of a user interactive deviceaccording to one embodiment.

FIG. 11 is a flow chart of a method implemented by a management devicewith respect to image recognition operations according to oneembodiment.

DETAILED DESCRIPTION

Attention is directed to the following commonly assigned applications,which are incorporated herein by reference:

U.S. patent application Ser. No. 12/847,754 entitled “Augmented Realityand Location Determination Methods and Apparatus” by inventors LukeRichey and Allen Greaves, and U.S. patent application Ser. No.12/847,790 entitled “Augmented Reality and Location DeterminationMethods and Apparatus” by inventors Luke Richey and Allen Greaves.

Some aspects of the disclosure described herein are directed towardsapparatus, methods and programming for implementing augmented realityoperations where the physical world is augmented with additionalinformation, such as virtual objects. For example, images of thephysical world observed through user interaction devices may beaugmented or enhanced with augmented reality representations, forexample in the form of visual and/or audio data which may be experiencedby users. In one example embodiment, augmented reality representationsmay include virtual objects which augment a user's experience of thephysical world. The virtual objects may be associated with physicalworld objects which may be static or dynamically moving. Some of thedescribed embodiments include a media system configured to implement andco-ordinate or manage augmented reality operations of one userinteraction device or a plurality of user interaction devices which maybe interacting in a collaborative augmented reality session in onearrangement.

Some augmented reality systems use location information regardinglocations of the user interaction devices and locations of physicalobjects in the physical world to accurately augment the physical worldwith the augmented reality representations. The location information maybe used to associate virtual objects with respective objects of thephysical world in one illustrative example. At least some aspects of thedisclosure are directed towards increasing the accuracy of generatedlocation information regarding the locations of user interaction devicesand which location information may be used to implement augmentedreality operations. At different times, a plurality of differenttechniques may be available to determine the location information ofuser interaction devices. In some embodiments, information from thedifferent techniques may be utilized and/or combined to provide locationinformation of the user interaction devices of increased accuracycompared with other available location information of the userinteraction devices, perhaps obtained from a single source. Additionalaspects are described in the following disclosure.

According to one embodiment, a location determination method includesaccessing first location information regarding a location of a userinteraction device in a physical world, wherein the user interactiondevice is configured to generate an augmented reality representationwith respect to the physical world, using the first locationinformation, identifying a plurality of wireless communication deviceswhich are proximately located with respect to the user interactiondevice, initiating wireless communications between the user interactiondevice and the wireless communications devices, after the initiating,accessing information regarding the wireless communications of the userinteraction device and the wireless communication devices, and using theinformation regarding the wireless communications, determining secondlocation information regarding the location of the user interactiondevice, and wherein the second location information has increasedaccuracy with respect to the location of the user interaction device inthe physical world compared with the first location information.

According to another embodiment, A location determination methodcomprises accessing first location information regarding a location of auser interaction device in a physical world, wherein the userinteraction device is configured to generate an augmented realityrepresentation with respect to the physical world, using the firstlocation information, identifying a marker which is proximately locatedwith respect to the location of the user interaction device, accessingan image generated by the user interaction device which includes themarker, and processing the image to determine second locationinformation regarding the location of the user interaction device, andwherein the second location information has increased accuracy withrespect to the location of the user interaction device in the physicalworld compared with the first location information.

According to yet another embodiment, an augmented reality methodcomprises accessing first location information regarding a location of auser interaction device in a physical world, wherein the userinteraction device is configured to generate an augmented realityrepresentation with respect to the physical world, using the firstlocation information, generating second location information which hasincreased accuracy regarding the location of the user interaction devicein the physical world, and communicating augmented data to the userinteraction device, and wherein the augmented data comprises theaugmented reality representation.

According to another embodiment, a location determination methodcomprises using a user interaction device, emitting a wirelesscommunications signal at a first moment in time, using a plurality ofwireless communication devices, receiving the wireless communicationssignal emitted by the user interaction device at a plurality of secondmoments in time, and using the first and second moments in time,determining information regarding a location of the user interactiondevice.

According to another embodiment, a computing system comprisescommunications circuitry configured to implement communicationsexternally of the computing system, and processing circuitry coupledwith the communications circuitry, and wherein the processing circuitryis configured to access first location information regarding a locationof a user interaction device in a physical world, to use the firstlocation information to identify a plurality of wireless communicationsdevices which are proximately located with respect to the location ofthe user interaction device, to control the communications circuitry tooutput a control signal which is configured to initiate wirelesscommunications between the user interaction device and the wirelesscommunications devices, to access information regarding the wirelesscommunications between the user interaction device and the wirelesscommunications devices, and to use the information regarding thewireless communications to determine second location informationregarding the location of the user interaction device and which hasincreased accuracy with respect to the location of the user interactiondevice in the physical world compared with the first locationinformation.

According to another embodiment, a computing system comprisescommunications circuitry configured to implement communicationsexternally of the computing system and processing circuitry coupled withthe communications circuitry, and wherein the processing circuitry isconfigured to access first location information received by thecommunications circuitry regarding a location of a user interactiondevice in a physical world, to identify a marker which is proximatelylocated with respect to the location of the user interaction device, toaccess a plurality of images generated by the user interaction device,and to process the images with respect to the marker to determine secondlocation information regarding the location of the user interactiondevice and which has increased accuracy with respect to the location ofthe user interaction device in the physical world compared with thefirst location information

According to another embodiment, a computing system comprisescommunications circuitry configured to implement communicationsexternally of the computing system and processing circuitry coupled withthe communications circuitry, and wherein the processing circuitry isconfigured to access first location information received by thecommunications circuitry regarding a location of a user interactiondevice in a physical world, to generate second location informationwhich has increased accuracy regarding the location of the userinteraction device in the physical world, and to control thecommunications circuitry to communicate augmented data comprising anaugmented reality representation to the user interaction device.

According to another embodiment, an augmented reality user interactiondevice comprises a camera, a display system, communications circuitryconfigured to implement wireless communications externally of the userinteraction device, and processing circuitry coupled with the camera,the display system, and the communications circuitry, wherein theprocessing circuitry is configured to control the display system togenerate a plurality of images which comprise image data generated bythe camera and augmented data which augments the image data with anaugmented reality representation, wherein the processing circuitry isfurther configured to control the communications circuitry tocommunicate first location information regarding a location of the userinteraction device in a physical world externally of the userinteraction device, and wherein the processing circuitry is furtherconfigured to access second location information regarding the locationof the user interaction device after the outputting of the firstlocation information and to use the second location information togenerate the plurality of images, and wherein the second locationinformation has increased accuracy regarding the location of the userinteraction device in the physical world compared with the firstlocation information.

Referring to FIG. 1, one example of augmented reality aspects of thedisclosure is described. FIG. 1 illustrates a user interaction device 10which is used to generate an image of the physical world and which isaugmented by an augmented reality representation. More specifically, inthe example of FIG. 1, the user interaction device 10 includes a camera(not shown) which is configured to capture images of the physical worldand which may depicted using a display 12. As a user moves the userinteraction device 10, a plurality of images are captured of differentscenes viewed by the camera of the device 10.

In the illustrated example, the scene viewed by the device 10 includes amarker 14 on a wall of the physical world. The generated image depictedusing the display 12 includes an augmented reality representation 18which augments a user's experience of the physical world by replacingthe physical world marker 14 with the representation 18. In theillustrated example, the augmented reality representation 18 is avirtual 3D object in the form of a puppy, which may be selected byanother user to be associated with the marker 14.

The use of marker 14 is one example of augmented reality operationswhich may be implemented using the user interaction device 10 and otheraugmented reality operations may be implemented in other embodiments.For example, virtual objects may be associated with other physicalobjects of the physical world, such as other user interaction devices 10(not shown), in images generated by device 10. In some embodiments,augmented reality representations 18 may entirely replace physicalobjects of the physical world.

In one more specific example, the augmented reality representations 18may include advertising objects (e.g., banner with a product name) andthe representations 18 may be associated with famous physical structuresof the physical world when observed through a user interaction device10. For example, a user at a significant football game may view avirtual object banner draped between the physical world goalposts when auser of a device 10 captures images of the end zone during a footballgame. Companies may pay advertising fees to have augmented realityrepresentations of advertisements of their products associated withphysical world objects and which may be viewed by users using their userinteraction devices 10 who are proximately located to the physical worldobjects in one embodiment.

Location information regarding the locations of the user interactiondevice 10 and other physical objects in the physical world may be usedto generate augmented reality representations 18 in captured images. Inone example, the location information may be used to depict theaugmented reality representations 18 accurately associated with content(e.g., objects) of the physical world (e.g., other user interactiondevices, buildings, structures, mountains, etc.).

For a static physical object which does not move (e.g., marker 14),location information may be included with the augmented data whichdetermines where the augmented reality representations 18 are to bedisplayed with respect to content of the physical world when the staticphysical object is within the field of view of the camera. For portablephysical objects (e.g., user interaction devices), the augmented datamay be associated with an identifier of the portable physical object.Identification information of the portable physical object and locationinformation of the portable physical object may be used to determinewhen the portable physical object is present within the field of view ofthe camera and where augmented reality representations 18 associatedwith the portable physical object should be shown in generated images.Location information regarding the user interaction devices 10 and/orphysical objects may be used to accurately show the augmented realityrepresentations 18 associated with the user interaction devices 10and/or physical world objects in images generated by the userinteraction devices 10 in one embodiment.

In some embodiments, one user interaction device 10 may be present andthe user may be experiencing augmented reality representations withrespect to physical objects of the physical world. In other examples(e.g., FIG. 1), a plurality of users having user interaction devices 10may be present and proximately located to one another and experiencingaugmented reality representations with respect to one another in acollaborative session and/or physical world objects. The augmentedreality representations may be associated with user interaction devices10 of users and/or with physical objects. Users and user interactiondevices 10 may be free to enter and leave interactive augmented realitycollaborative sessions in some embodiments.

It is desired to provide accurate information regarding the locations ofuser interaction devices 10 to correctly associate augmented realityrepresentations with respect to physical world objects. As describedfurther below, methods and apparatus are described which enable locationinformation of the user interaction devices 10 to be determined withincreased accuracy compared with, for example, arrangements which useconventional location determination methods, such as a globalpositioning system (GPS). In some embodiments disclosed below, the userinteraction devices 10 may be configured to communicate (e.g.,wirelessly) with one another as well as with external devices (e.g., amanagement device, Wi-Fi communications devices) to implement augmentedreality operations including operations with respect to determininglocation information of the user interaction devices 10 of increasedaccuracy.

Referring to FIG. 2, one example of a media system 20 is shown. Mediasystem 20 is configured to implement operations with respect augmentingthe physical world with augmented reality representations. For example,media system 20 is configured to assist user interaction devices 10 withthe generation of augmented reality representations. In a more specificexample, media system 20 is configured to perform operations withrespect to determining locations of user interaction devices 10 (whichmay be portable) for use in accurately associating augmented realityrepresentations with the physical world. In one embodiment, media system20 may communicate augmented data with respect to the user interactiondevices 10 which may be used by the devices 10 to generate augmentedreality representations.

In the illustrated example configuration of FIG. 2, media system 20includes a plurality of use interaction devices 10 and a managementdevice 22. User interaction devices 10 may be configured to communicatewith one another as well as with management device 22. For example, theuser interaction devices 10 may communicate with management device 22via a network 24. Network 24 may be considered to be a part of mediasystem 20 or may be external of media system 20 in differentembodiments. In some embodiments, the user interaction devices 10 mayalso implement wireless communications with respect to other wirelesscommunications devices (e.g., Wi-Fi communications devices) which may bewithin the communications ranges of the devices 10 (the Wi-Ficommunications devices are not shown in FIG. 2).

The user interaction devices 10 may be proximately located with respectto one another in a group (e.g., within communications ranges of thedevices 10 to implement communications with respect to one another) orin different geographical locations and not proximately located to oneanother. For example, different groups of user interaction devices 10may exist in different geographical locations. User interaction devices10 which are proximately located to one another may participate in acollaborative augmented reality session where augmented realityrepresentations may be associated with the devices 10 in one embodiment.Additionally, only one user interaction devices 10 may be present in agiven geographical location and may be implementing augmented realityoperations with respect to static physical world objects.

User interaction devices 10 may be computing systems (e.g., one exampleis described with respect to FIG. 3) in one embodiment. The userinteraction devices 10 may have substantially the same configurations orhave different configurations in example embodiments. In some examples,user interaction devices 10 may be configured as portable media devices,personal digital assistants, cellular telephones, smartphones, personalcomputers, notebook computers, glasses worn by a user including a cameraand display system capable of generating images, or any other devicecapable of capturing images of the physical world and generating imagesand/or other media content for consumption by a user which includevisual images of the physical world which are augmented by one or moreaugmented reality representations (e.g., additional virtual imagecontent and/or audible content which augments physical world content).

In one embodiment, management device 22 may be a server which isconfigured as a computing system, for example as described below withrespect to FIG. 3. Management device 22 is configured to implementcommunications with respect to user interaction devices 10 in thedescribed embodiment. Management device 22 may be configured to performa plurality of operations with respect to the generation of augmentedreality representations by the user interaction devices. Exampleoperations performed include operations with respect to co-ordinationand management of user interaction devices 10, co-ordination andmanagement of communications between user interaction devices 10 and aplurality of other wireless communications devices, determininglocations of user interaction devices 10 (which may be portable), andstoring and communicating augmented data for use by the user interactiondevices 10 to generate augmented reality representations.

In one example embodiment, the user interaction devices 10 maycommunicate augmented data of their respective augmented realityrepresentations (e.g., the above-described puppy) to the managementdevice 22, perhaps for storage, and the management device 22 maythereafter provide the augmented data to others of the user interactiondevices 10 for use in generating the augmented reality representationswith respect to the devices 10 which provided the augmented data. Forexample, in FIG. 1, the management device 22 may communicate augmenteddata which includes the puppy representation to user interaction device10 which uses the augmented data to generate the augmented realityrepresentation 18.

In one embodiment, identification data may be used to associateaugmented data with respective appropriate objects of the physicalworld, such as user interaction devices 10 or other physical worldobjects. With respect to user interaction devices as described furtherbelow in one embodiment with respect to FIG. 1, initial locationinformation regarding the location of a user interaction device 10 maybe used to search a database of the management device 22 to identifyother wireless communications devices (e.g., user interaction devices,Wi-Fi communications devices) within a communications range of the userinteraction device 10. The database may include location informationregarding the devices 10 which the initial location information issearched against, and identification information which uniquelyidentifies each of the wireless communications devices.

The database may also include identification information whichidentifies the augmented data associated with the user interactiondevices 10. In one example, once the management device 22 determinesthat another user interaction device is proximately located to anotheruser interaction device 10, the management device 22 may communicate theaugmented data associated with one of the devices to the other deviceand which includes an identifier of the device to which the augmentedreality representation of the augmented data is to be associated with.The receiving user interaction device may access location informationregarding its present location and the location of the other device(e.g., access the location information from management device 22) anduse the location information of the devices, the identificationinformation of the other device, and the respective augmented data ofthe other device to generate the augmented reality representation 18associated with the other device 10 in images captured by the devicewhich include the other device.

In one embodiment, the device 10 may also depict other augmented realityrepresentations associated with other proximately located userinteraction devices 10 and perhaps other objects of the physical worldwhich may be present and have associated augmented data. Morespecifically, some objects of the physical world are static and do notchange locations (e.g., marker 14 of FIG. 1). In one embodiment,management system 22 may store augmented data to be used to generateaugmented reality representations with respect to static physicalobjects and the management system 22 may use location informationregarding the device 10 to search the database to identify augmenteddata for static physical objects which are proximately located to thelocation of the device 10. The management device 22 may communicateaugmented data for a static physical object which is proximately locatedto the device 10 to the device 10 as well as location information wherethe static physical object exists in the physical world. Thereafter, thedevice 10 may use the location information to determine when the staticphysical object is within the field of view of the camera of device 10and the device 10 may augment captured images of the static physicalobject with augmented reality representations using the appropriateaugmented data.

Additional operations of the user interaction devices 10 with respect togeneration of augmented reality representations may be managed bymanagement device 22 in one embodiment. In one example managementembodiment, user interaction devices 10 which attempt to access andutilize media system 20 to experience or implement operations withrespect to augmented reality may be managed by management device 22. Inone embodiment, management device 22 may manage an augmented realitysession where a plurality of user interaction devices 10 are present andperforming augmented reality operations with respect to one another.

In one example, media system 20 may be a closed system and userinteraction devices 10 may first be registered and/or otherwiseauthorized prior to having permission to gain access to media system 20.In one embodiment, management device 22 may assign passwords to thedevices 10 and use the passwords and identification information of thedevices 10 to verify that the user interaction devices 10 are authorizedto gain access to the media system 20. In some arrangements, users mayinitially sign-up to gain access to the media system 20, and afterreceiving a password, may subsequently participate in augmented realitysessions or otherwise access the media system 20 to implement augmentedreality operations. Users may pay fees to participate in servicesprovided by media system 20, for example, including services ofaccessing management device 22 and implementing augmented realityoperations with respect to other user interaction devices 10 managed bymanagement device 22. Access to or participation in augmented realityoperations provided by media system 20 and management device 22 may bedenied to users who cannot provide appropriate identification andpassword information in one embodiment.

As described in detail below in some embodiments, management device 22may perform operations with respect to determination of locationinformation of the user interaction devices 10 which may be used toimplement augmented reality operations with respect to the userinteraction devices 10. Some of the described location determinationembodiments include implementing wireless communications between theuser interaction devices 10 and/or other wireless communications deviceswhich may be present to determine the location information. Managementdevice 22 may be configured to manage and/or coordinate at least some ofthe communications of the user interaction devices 10 in one example. Inaddition, management device 22 may be configured to implement operationswith respect to image processing to determine the location informationin additional embodiments described below. Other operations may beperformed by management device 22 with respect to augmented reality andsome of the operations performed by the management device 22 may beperformed by other devices (e.g., the user interaction devicesthemselves) in other embodiments.

Network 24 which may be implemented as any suitable networkconfiguration for implementing wired and/or wireless communicationsbetween user interaction devices 10, management device 22 or otherwireless communications devices. For example, network 24 may include awireless network, local or wide area networks, Internet, cellularnetwork, and/or other suitable infrastructure for implementingcommunications of digital information. In one embodiment, userinteraction devices 10 may have cellular communications capabilities tocommunicate with network 24 and management device 22. Network 24 mayimplement packet-switched communications between user interactiondevices 10 and management device 22 in one embodiment. As mentionedabove, user interaction devices 10 may also communicate directly withone another or other wireless communications devices without use ofnetwork 24 in one embodiment.

Referring to FIG. 3, one example embodiment of a computing system 30 isshown. The illustrated system 30 includes communications circuitry 32,processing circuitry 34, storage circuitry 36, a user interface 38, anda camera 40. Other embodiments of computing system 30 are possibleincluding more, less and/or alternative components. For example, theillustrated configuration of computing system 30 may correspond to oneof user interaction devices 10 while some of the illustrated components(e.g., camera 40) may be omitted in one arrangement of computing system30 which is implemented as management device 22.

Communications circuitry 32 is arranged to implement communications ofcomputing system 30 with respect to external devices or systems (e.g.,network 24 and other computing systems 30 implemented as other userinteraction devices 10, Wi-Fi communications devices, or managementdevice 22). Communications circuitry 32 may be configured to implementwired and/or wireless communications. Additional details of one exampleof communications circuitry 32 which may be utilized is discussed belowwith respect to FIG. 4.

In one embodiment, processing circuitry 34 is arranged to process data,control data access and storage, issue control signals or commands, andcontrol other augmented reality operations. Processing circuitry 34 maycomprise circuitry configured to implement desired programming providedby appropriate computer-readable storage media in at least oneembodiment. For example, the processing circuitry 34 may be implementedas one or more processor(s) and/or other structure configured to executeexecutable instructions including, for example, software and/or firmwareinstructions. Other exemplary embodiments of processing circuitry 34include hardware logic, PGA, FPGA, ASIC, state machines, and/or otherstructures alone or in combination with one or more processor(s). Theseexamples of processing circuitry 34 are for illustration and otherconfigurations are possible.

Storage circuitry 36 is configured to store programming such asexecutable code or instructions (e.g., software and/or firmware),electronic data, databases, image data, augmented data, identifiers,location information and/or other digital information and the storagecircuitry 46 may include computer-readable storage media. At least someembodiments or aspects described herein may be implemented usingprogramming stored within one or more computer-readable storage mediumof storage circuitry 36 and configured to control appropriate processingcircuitry 34.

The computer-readable storage medium may be embodied in one or morearticles of manufacture which can contain, store, or maintainprogramming, data and/or digital information for use by or in connectionwith an instruction execution system including processing circuitry 34in the exemplary embodiment. For example, exemplary computer-readablestorage media may include any one of physical media such as electronic,magnetic, optical, electromagnetic, infrared or semiconductor media.Some more specific examples of computer-readable storage media include,but are not limited to, a portable magnetic computer diskette, such as afloppy diskette, a zip disk, a hard drive, random access memory, readonly memory, flash memory, cache memory, and/or other configurationscapable of storing programming, data, or other digital information.

User interface 38 is configured to interact with a user includingconveying data to a user (e.g., displaying visual images for observationby the user) as well as receiving inputs from the user, for example, viaa graphical user interface (GUI). User interface 38 may be configureddifferently in different embodiments. One example embodiment of userinterface 38 is discussed below with respect to FIG. 5.

Camera 40 is configured to capture images within its field of view andgenerate image data of scenes of the physical world viewed by thecomputing system 30 in one embodiment. An example camera 40 includes anappropriate imaging sensor configured to generate digital image dataresponsive to received light in one implementation.

Movement/orientation circuitry 42 is configured to provide informationregarding movement and orientation of the computing system 30 in thedescribed embodiment. For example, circuitry 42 may include anaccelerometer arranged to provide information regarding forces which thecomputing system is subjected to. Circuitry 42 may also include acompass and inclinometer configured to provide information regarding anorientation of the computing system 30 in the physical world.

Referring to FIG. 4, communications circuitry 32 of computing system 30which is implemented as one of user interaction devices 10 is shownaccording to one possible embodiment. In one embodiment, communicationscircuitry 32 is configured to implement wired and/or wirelesscommunications. In the illustrated configuration, communicationscircuitry 32 includes network communications circuitry 50, GPScommunications circuitry 52, and local communications circuitry 54.Other configurations of communications circuitry 32 are possibleincluding more, less and/or alternative components. For example, acommunications interface (e.g., USB port, NIC interface) forimplementing wired communications may also be provided.

In one embodiment, network communications circuitry 50 is configured toimplement wireless communications with respect to network 24. In onemore specific embodiment, network communications circuitry 50 isconfigured to communicate with cell towers of network 24 which includesa wireless cellular network. Accordingly, network communicationscircuitry 50 may implement wireless communications over relatively largedistances (e.g., miles) in at least one embodiment.

GPS communications circuitry 52 is configured to implement wirelesscommunications with external GPS transmitters (e.g., satellites). In oneembodiment, GPS communications circuitry 52 may also include appropriatecircuitry to provide Assisted GPS (AGPS) or Wireless Assisted GPS(WAGPS) which utilizes additional location information to providelocation information of increased accuracy compared with use of GPSalone. GPS communications circuitry 52 may provide location informationof computing system 30 directly in one embodiment or processingcircuitry 34 may be configured to process signals received by GPScommunications circuitry 52 to provide the location information of thecomputing system 30 in another embodiment.

Local communications circuitry 54 is configured to implement wirelesscommunications with respect to local wireless communications devices(e.g., other user interaction devices 10, Wi-Fi communications devices)which are proximately located to the computing system 30. In oneembodiment, local communications circuitry 54 is configured to implementBluetooth, ad hoc or other suitable wireless communications with respectto the local wireless communications devices.

Referring to FIG. 5, one example embodiment of a user interface 38 of auser interaction device 10 is shown. The example user interface 38includes an input device 60, display system 62, speaker 64 andmicrophone 66. Other arrangements of user interface 38 are possibleincluding more, less and/or additional components.

Input device 60 is arranged to receive inputs of a user interacting withcomputing system 30. Input device 60 may include a graphical userinterface, keyboard, pointing device (e.g., mouse) or other suitableapparatus to permit a user to input information and interact with thecomputing system 30.

Display system 62 is configured to generate visual images which may beviewed by the user. Display system 62 may depict images captured bycamera 40, display digital content (e.g., windows, email, etc.) anddisplay augmented reality representations in some examples.

Speaker 64 is configured to emit sound waves in one embodiment. Thesound waves may include voice information, audio information and othersound waves which are discussed further below.

Microphone 66 is a configured to receive sound waves in one embodiment.Received sound waves may include voices and sound waves emitted by othercomputing systems 30 in some illustrative examples.

As discussed previously, user interaction devices 10 are configured toimplement operations to enable users of the devices 10 to experienceaugmented reality. In one embodiment, the user interaction devices 10may participate in augmented reality sessions which include a pluralityof the devices 10. In another embodiment, a single user interactiondevice 10 may implement operations for its user to experience augmentedreality without the presence of other user interaction devices 10 (e.g.,associating augmented reality representations with physical objectspresent in the physical world). In some embodiments, the userinteraction devices 10 may access management device 22 which mayimplement, manage and/or co-ordinate augmented reality operations for asingle user interaction device 10 or a plurality of user interactiondevices 10 (which may be participating in an augmented realitycollaborative session with one another).

Accurate location information regarding locations of one or more userinteraction device 10 is desired to improve the experience of augmentedreality by the users. For example, the location information may be usedby the user interaction device 10 to correctly and accurately associateaugmented reality representations with respect to physical objects ofthe physical world. The augmented reality representations may becorrectly associated with physical objects of the physical world whensufficiently accurate location information regarding the locations ofthe user interaction devices 10 and physical objects are used toimplement augmented reality operations. Less accurate informationregarding the locations of user interaction devices 10 may result inless than desirable augmented reality experiences. For example, anaugmented reality symbol, such as a puppy, which is to be associatedwith a respective user and user interaction device 10 may be associatedwith improper user interaction devices, improper physical world objects,or otherwise not correctly depicted as being associated with theappropriate user interaction device 10.

Furthermore, in some embodiments, the physical objects which augmentedreality representations are to be associated with may be portable (e.g.,user interaction devices 10) whose locations may be constantly changingwhile augmented reality operations are being implemented. The locationsof the portable physical objects are determined in sufficient real timeto enable the augmented reality representations to be appropriateassociated with the portable physical objects (i.e., enable theaugmented reality representations to be depicted near or upon theappropriate physical objects in real time). Additionally, portablephysical objects may move indoors and it is also desirable to enableaugmented reality operations to be implemented indoors or at otherlocations where location information of the portable physical objectsobtained via conventional sources (e.g., GPS) may not be sufficientlyaccurate.

In one embodiment, media system 20 including user interaction devices 10and management device 22 may be arranged to implement locationdetermination operations to provide sufficiently accurate locationinformation regarding the user interaction devices 10 and other physicalobjects of the physical world to provide satisfactory experiences ofaugmented reality to users of the user interaction devices 10. Aplurality of different methods for providing location information of thephysical objects may be available for use. In some arrangements, thelocation results of the different methods may be combined to furtherimprove the accuracy of the location information. In other arrangements,less than all of the methods may be available for use, but the availablemethods may provide improved results of increased accuracy of thelocations of the user interaction devices 10 or other physical objectscompared with reliance upon conventional methodologies (e.g., GPS).

Referring to FIG. 6, one example process of determining and usinglocation information of one or more user interaction devices 10 isshown. In one embodiment, the method may be implemented by processingcircuitry of a user interaction device which has been authorized toparticipate in augmented reality operations of the media system.Additional methods are possible including more, less and/or alternativeacts.

At an act A10, a user interaction device may determine initial locationinformation regarding its present location at an initial moment in time,for example when the user interaction device accesses the media systemto implement augmented reality operations. In one example, the userinteraction device may use location determination circuitry to providethe location information using aGPS. The initial location informationdetermined by the user interaction device may have unacceptable error toimplement accurate augmented reality operations with respect to thephysical world depending upon the location of the user interactiondevice (e.g., indoors or otherwise located where reception of GPScommunications may be relatively poor). Other techniques may also beutilized, for example, monitoring movements of the user interactiondevice from a last known position, for example, using output ofmovement/orientation circuitry 42.

At an act A12, the user interaction device may communicate the initiallocation information to the management device of the media system. Inone embodiment, the initial location information may be aGPS data withan associated error reading. The user interaction device may alsoinclude a timestamp corresponding to the time when the locationinformation was obtained and identification information which uniquelyidentifies the user interaction device.

The management device of the media system may utilize the initiallocation information to perform operations to attempt to determinerefined location information which has increased accuracy with respectto the actual location of the user interaction device in the physicalworld compared with the initial location information. As discussedbelow, the management device may manage communications of the userinteraction device with respect to other wireless communications devices(e.g., other user interaction devices and Wi-Fi communications devices)to provide refined location information according to one method and/orimplement image recognition operations to provide the refined locationinformation according to another method in illustrative exampleembodiments.

At an act A14, the user interaction device monitors for the reception ofa communication from the management device. The communication indicateswhether a sufficient number of wireless communications devices areproximately located to the user interaction device to implementoperations with respect to the devices to attempt to provide the refinedlocation information or the communication indicates that an insufficientnumber of wireless communications devices are present to implement theoperations. In one embodiment, a threshold distance (e.g., correspondingto a local wireless communications range of the user interaction devicewhich provided the initial location information in one example) may beused to determine whether other wireless communications devices arelocated sufficiently close to the communicating user interaction device.In one embodiment, the management device may maintain a database oflocations of the wireless communications devices and the managementdevice may use the initial location information to search the databaseto identify the presence of other wireless communications devices whichmay be proximately located to the user interaction device to implementwireless communications with the user interaction device.

If the result of act A14 is negative, the user interaction device mayenter a device waiting state or mode as a waiter as described furtherbelow. In addition, the method may proceed to an act A20 to implementadditional operations with respect to image recognition as discussed indetail below.

If the result of act A14 is affirmative, the process proceeds to an actA16 where the user interaction device implements localizedcommunications with respect to the other proximately located wirelesscommunications devices which were identified to be sufficiently close tothe user interaction device for wireless communications. According toone example embodiment, the user interaction device may be controlled tobe in different operational states as a transmitter and receiver ofwireless communications signals at different moments in time toimplement the wireless communications. Additional details of thesecommunications and different operational states of the user interactiondevices are discussed below with respect to the example embodiments ofFIGS. 7-9.

At an act A18, information regarding the wireless communications betweenthe proximately-located wireless communications devices may becommunicated by the wireless communications devices participating in thecommunications to the management device. In one embodiment, themanagement device may process the information regarding the wirelesscommunications in an attempt to provide refined location information forthe wireless communications devices including the user interactiondevice which provided the initial location information at act A10.Additional details regarding the processing to determine the refinedlocation information using the information regarding the wirelesscommunications is described in further detail below with respect to oneexample embodiment of FIGS. 10 a-10 c.

In some embodiments where communications are implemented with respect toone or more Wi-Fi communications devices which are proximately locatedwith respect to the user interaction device, the locations of the one ormore Wi-Fi communications devices may be fixed and the known locationsof these static communications devices may also be used to determine thelocation information of the user interaction device. The usage of thelocation information of these devices may provide increased accuracycompared to communications which do not utilize a static device sincethe location information of the static device may be known with arelatively high accuracy compared with locations of portable devices.

Following the communication of the information regarding the wirelesscommunications, the user interaction device which provided the initiallocation information may proceed to implement operations with respect toimages in an attempt to determine additional information regarding itslocation as described further below.

At an act A20, the user interaction device monitors for the reception ofa communication from the management device indicating that one or moremarkers are proximately located to the user interaction device (e.g.,within visible range of the optics of the camera user interactiondevice). For example, the management device may maintain a database ofvisual markers and their respective locations in the physical world andthe management device may use the initial location information (as wellas other location information resulting from the wirelesscommunications) to search the database to identify markers which arelocated sufficiently close to the user interaction device to be capturedby the camera of the user interaction device.

If the result of act A20 is negative, the process proceeds to an act A26where the location information for the user interaction device may beupdated, for example, using refined location information from themanagement device (e.g., based upon the wireless communications withother wireless communications devices). The refined location informationmay be stored as the location of the respective user interaction devicein a database maintained by the management device and the refinedlocation information may be communicated to one or more of the userinteraction devices.

If the result of act A22 is affirmative, the process proceeds to an actA22 where the user interaction device which provided the initiallocation information captures images of the physical world about theuser interaction device.

At an act A24, the images may be processed by the user interactiondevice in attempt to provide the refined location information. In oneembodiment, the management device may communicate images of the markers(e.g., as well as information regarding locations in the physical worldfrom which the images were captured) which are proximately located tothe user interaction device to the user interaction device. The markersmay be displayed to a user to inform the user of the presence and typesof nearby markers.

The user interaction device may implement image processing techniques toanalyze images captured by the user interaction device to identifywhether the markers are present in the images. For example, objectrecognition processing may be used in one arrangement to identify themarkers. Once the markers are identified, the images captured by theuser interaction device may be processed with respect to the images ofthe markers received by the management device to identify locationinformation regarding the locations of the user interaction device fromwhich the images of the markers were captured by the user interactiondevice. This analysis may compare differences of the images and use thecomparison with the known location information from which the images ofthe markers provided by the management device were captured. Thisdetermined location information may have increased accuracy of thelocation of the user interaction device compared with other availablelocation information regarding the location of the user interactiondevice. Additional details regarding processing of images captured bythe user interaction device are described below with respect to FIG. 11in one embodiment.

In another embodiment, image data of images captured by the userinteraction device may be communicated to the management device forprocessing to provide the determined location information. In someembodiments, the markers which are in the vicinity of the userinteraction device may be communicated to the user interaction device toinform the user of available markers and thereafter the user may captureimages of the markers and communicate the captured images to themanagement device for processing as described above in act A24. In otherarrangements, the markers may not be communicated to the userinteraction device and the user interaction device may upload capturedimages to the management device for processing to recognize the presenceof any markers and determine the location information. Other methods arepossible.

At an act A26, the refined location information determined by thewireless communications with other wireless communications devicesand/or information obtained by image processing techniques may be usedto update the location information of the user interaction device. Therefined location information may have increased accuracy of the locationof the user interaction device in the physical world compared with otheravailable location information (e.g., GPS signals received by the userinteraction device located in an indoors environment).

At an act A28, the user interaction device and/or management device mayuse the refined location information for various purposes. In oneexample, the user interaction device and/or management device mayutilize the refined location information to implement augmented realityoperations. In one more specific example, the user interaction devicemay utilize the refined location information to generate images whichinclude augmented reality representations associated with physical worldcontent. For example, the refined location information may be used todetermine where the augmented reality representations will be depictedin the generated images of the physical world and to be viewed in realtime by the user of the user interaction device in one embodiment. Themanagement device may use the refined location information of a userinteraction device 10 to identify markers or other user interactiondevices which may be proximately located to the refined locationinformation of the user interaction device 10.

As mentioned above, a user interaction device 10 may wirelesslycommunicate with other wireless communications devices which may beproximately located to the device 10 in attempts to determine refinedlocation information regarding the actual location of the device 10 inthe physical world with increased accuracy compared with other availablelocation determination methods. An example implementation of usingwireless communications to determine refined location information isdescribed in additional detail with respect to FIGS. 7-9. The methodsmay be performed to determine refined location information usingwireless communications between the wireless communications devices(e.g., user interaction devices, Wi-Fi communications devices) which arewithin wireless communications range of one another. More specifically,the method of FIG. 7 controls the states or modes of operation of thewireless communications devices as waiters, senders and receivers in oneembodiment. The method of FIG. 8 discloses one method of operations withrespect to one of the devices operating as a sender and the method ofFIG. 9 discloses one method of operations with respect to one or more ofthe devices operating as receivers. The wireless communications devicesmay individually operate as a waiter, sender and receiver at differentmoments in time. At one moment in time, one of the wirelesscommunications devices may operate as a sender to transmit a wirelesscommunications signal and the other devices operate as receivers of thesignal. At subsequent moments in time, the other wireless communicationsdevices may individually operate as senders while the non-transmittingones of the devices operate as receivers of the transmissions.

Referring to FIG. 7, a method of controlling the states or modes ofoperation of the wireless communications devices is described accordingto one embodiment. Other methods are possible including more, lessand/or alternative acts. The method may be executed by processingcircuitry of the management device in one implementation.

As mentioned above, the management device manages the wirelesscommunications in one embodiment. In one illustrative example, aplurality of wireless communications devices may be within a wirelesscommunications range of one another and the devices may implementwireless communications with respect to one another. The wirelesscommunications may be used to determine refined location information ofthe devices with increased accuracy compared with the initial locationinformation (e.g., GPS data) of the devices. The wireless communicationsdevices may communicate initial location information and timeinformation to the management device 22 and the management device 22 mayuse the location information and time information to search a databaseto identify the wireless communications devices which are within rangeof one another and capable of communicating with one another.

In one example communications arrangement discussed below, themanagement device determines the states or modes of operations wirelesscommunications devices at different moments in time during the wirelesscommunications. As discussed above in one example, the management devicemay instruct one of the wireless communications to operate as a senderof a wireless communications signal at one moment in time and instructothers of the wireless communications devices to operate as receivers toreceive the wireless communications signal. At other moments in time,the management device may instruct others of the wireless communicationsdevices to individually operate as the senders and the non-transmittingdevices may be instructed to operate as receivers. After each of thewireless communications devices has operated as a sender, the results ofthe wireless communications may be analyzed to provide refined locationinformation regarding the locations of the wireless communicationsdevices in the physical world. When an insufficient number of wirelesscommunications devices are present (or at other appropriate moments intime), the management device may instruct the wireless communicationsdevices to operate as waiters where no wireless communications areimplemented until a sufficient number of wireless communications (e.g.,three) are within communications range of one another for the locationdetermination operations to occur.

At an act A100, initial location information is accessed from a userinteraction device. In one embodiment, the user interaction devicecommunicates GPS location coordinates, accuracy information regardingthe GPS data regarding the location of the user interaction device and aunique identifier to the management device. The user interaction devicemay also communicate accelerometer information (i.e., if the userinteraction device is configured with an on-board accelerometer) whichmay be used to modify the initial location information by an offsetaccording to movement detected by the accelerometer. A timestampregarding the initial location information may also be recorded in oneembodiment. The timestamp and initial location information for the userinteraction device may be stored within a database of the managementdevice as discussed below in one embodiment.

At an act A102, the processing circuitry accesses the database of themanagement device including information regarding wirelesscommunications devices. The database may include respective locationinformation for a plurality of user interaction devices which areinteracting with the management device, and perhaps implementingaugmented reality operations. Furthermore, the database may alsomaintain location information for a plurality of additional wirelesscommunications devices, such as Wi-Fi communications devices.

At an act A104, it is determined whether the mapping is empty for theuser interaction device which communicated the initial locationinformation.

If the condition of act A104 is affirmative, a new mapping entry iscreated for the user interaction device and the received initiallocation information is stored as the location of the device at an actA106.

If the condition of act A104 is negative, the timestamp of the initiallocation information is compared with the last modification of the entryfor the user interaction device at an act A108.

At an act A110, it is determined whether the length of time from thelast entry to the new information exceeds a threshold.

If the condition of act A110 is affirmative, the mapped entry for theuser interaction device is reset with the received initial locationinformation at an act A112.

If the condition of act A110 is negative, the received initial locationinformation may be disregarded and the stored location information forthe user interaction device may be used.

At an act A114, it is determined whether any other wirelesscommunications devices are proximately located to the location of theuser interaction device. As mentioned above, the management device maymaintain a database including location information for a plurality ofwireless communication devices (e.g., user interaction devices, Wi-Ficommunications devices). The management device may search the databaseto determine whether a sufficient number of wireless communicationsdevices are within a threshold distance (e.g., wireless communicationsrange) of the user interaction device. It is desired that at least threewireless communications devices be able to send and receive wirelesscommunications with respect to one another in one embodiment. Theaccuracy of the location information determined from wirelesscommunications of the devices increases as the number of communicatingdevices within communications range of one another increases.Furthermore, the accuracy of the information is increased if one or moreof the wireless communications devices have a known static location inthe physical world. If a sufficient number of devices were not locatedin act A114, the management device may output a control signal toinstruct the user interaction device at an act A116 to become a waiteruntil a sufficient number of devices are proximately located to the userinteraction device, for example, as determined by the management device.

If a sufficient number of devices were located in act A114, themanagement device determines whether the user interaction device hasrecently operated as a sender at an act A118. For example, themanagement device may determine whether the user interaction device haspreviously operated as a sender with the other wireless communicationsdevices which are currently within the communications range of the userinteraction device.

If the result of act A118 is affirmative, the management device proceedsto an act A122 to determine whether any of the other wirelesscommunications devices which are proximately located to the userinteraction device may become a sender.

If the result of act A118 or act A122 is negative, the management deviceoutputs a control signal instructing the user interaction device whichprovided the initial location information to become a sender at an actA124. As discussed further below, the user interaction device may outputa wireless communications signal which may be received by other wirelesscommunications devices operating as receivers and the results of thecommunication may be used to provide refined location informationregarding the location of the user interaction device in one embodiment.

If the result of act A122 is affirmative, the management device mayoutput a control signal at an act A126 instructing the user interactiondevice to be a receiver which receives wireless communications signalsfrom senders as discussed in detail below. In addition to outputting thecontrol signals to the sender and receivers, the management device mayalso output a common timing reference to the senders and receivers whichthe senders and receivers may use to timestamp moments in time when thewireless communications signal is transmitted and received. For example,the timing reference may be the current time as determined by themanagement device and the senders and receivers may align their internaltiming references to this time in one embodiment.

At an act A128, the management device may implement operations withrespect to the user interaction devices. For example, the managementdevice may communicate with the user interaction devices operating assenders and receiver in the example embodiments of FIGS. 8 and 9.

Referring to FIG. 8, an example method of operations implemented by awireless communications device (e.g., user interaction device) operatingas a sender is described. Other methods are possible including more,less and/or alternative acts.

As mentioned above, wireless communications between a plurality ofwireless communications devices may be used to provide refined locationinformation regarding the wireless communications devices. In oneembodiment, one of the wireless communications devices operating as asender emits a wireless communications signal which may be received byother wireless communications devices within a wireless communicationsrange of the sender. The sender may output a plurality of differenttypes of wireless communications signals in the described arrangement.

More specifically, in one example embodiment, the sender emits anelectromagnetic wireless communications signal (e.g., Bluetooth, ad hocwireless communications, or other signal). In another exampleembodiment, the sender emits a sound wave (e.g., a sound wave having afrequency which may be outside of the audible range of humans). Thecommunicated signals may be received by the receivers and used toprovide refined location information in one embodiment. In someembodiments, the wireless communications devices may communicate pluraldifferent types of signals (e.g., electromagnetic signals as well assound waves).

At an act A150, a user interaction device which has been selected to bea sender communicates initial location information regarding its currentlocation which is received by the management device.

At an act A152, the management device accesses the initial locationinformation.

At an act A154, the management device creates a unique identifier forthe user interaction device which acts as the sender. If sound waves areto be communicated, the management device may select (e.g., randomly inone embodiment) a unique frequency from a range of possible frequenciesas the identifier which may be used. In other examples, the identifiermay be an oscillation of sound frequencies or a Bluetooth server namewhich the device uses to transmit a Bluetooth signal.

At an act A156, the management device searches for other wirelesscommunications devices which are proximately located to the userinteraction device to act as receivers (if the receivers are not known)or the management device accesses a list of the proximately locateddevices which may act as receivers (if known).

At an act A158, the management device communicates the unique identifierof the sender to the receivers. The unique identifier may also includesignal identification information which enables the receivers toidentify the communications from the sender (e.g., identifies the senderof a Bluetooth communications signal, identifies the frequency of asound wave from the sender, etc.).

At an act A160, the management device receives a plurality of responsesfrom the receivers indicating that they have received the uniqueidentifier and signal identification information and they are ready toreceive the wireless communications signal to be emitted from thesender.

At an act A162, the management device communicates the unique identifierto the sender indicating that the receivers are ready and the managementdevice may communicate a control signal to initiate wirelesscommunications. The management device may also specify the frequency ofa sound wave to be emitted if the devices are using sound waves.

At an act A164, the sender outputs the wireless communications signal(e.g., electromagnetic wave or sound wave) and records the time thesignal is outputted. For electromagnetic signals, the sender may useappropriate communications circuitry to output the signal. For soundwaves, a speaker of the sender may be used to output the signal.

At an act A166, the timestamp indicative of the time that the signal wasoutputted by the sender is communicated to the management device.

At an act A168, the management device accesses the timestamp regardingthe outputting of the signal.

At an act A170, the management device accesses a plurality ofcommunications from the receivers indicating the respective times thewireless communications signal which was outputted by the sender wasreceived by the respective receivers. The wireless communications signaloutputted by the sender may be received by the receivers at differenttimes depending upon their distances from the sender.

At an act A172, the management device uses the information regarding thelengths of time between the outputting of the wireless communicationssignal and the reception of the signal by the plural receivers todetermine linear distances between the sender and the receivers usingthe respective lengths of time of communications with respect to thereceivers which may be multiplied by the velocity of the wirelesssignals which were communicated (e.g., sound, electromagnetic). Thedetermined linear distances may be used to determine refined locationinformation of the user interaction devices as discussed in oneembodiment below with respect to FIGS. 10 a-10 c. The above-describedprocess may be repeated a number of times between two devices and theresultant distances may be averaged in one embodiment.

The management device may also output a completion signal to thewireless communications devices indicating that it has obtained allinformation regarding the communication (i.e., transmission andreception) of the wireless communications signal. The completion signalmay instruct the sender to cease sending the wireless communicationssignal.

At an act A174, the sender receives the completion signal from themanagement device and ceases outputting of the wireless communicationssignal.

At an act A176, the sender outputs a query to determine the next desiredstate of operation of the wireless communications device as a receiveror waiter.

At an act A178, the sender receives a response to the query from themanagement device in the form of a control signal to enter anoperational state as a receiver or a waiter.

At an act A180, the sender changes its operational state to be areceiver for subsequent communications from other proximately locatedwireless communications devices or a waiter in accordance with thereceived response to the query.

Referring to FIG. 9, an example method of operations implemented by awireless communications device (e.g., user interaction device) operatingas a receiver is described. Other methods are possible including more,less and/or alternative acts.

At an act A200, the user interaction device receives a unique identifierfrom the management device which identifies wireless communications tobe received from the sender. The unique identifier may identify thesender of Bluetooth signal or may identify a frequency of a sound waveand the sender of the sound wave.

At an act A202, the receiver prepares to receive the signal andcommunicates a ready signal to the management device.

At an act A204, the receiver waits for reception of the wirelesscommunications signal to be communicated by the sender.

At an act A206, the receiver compares an amount of time waiting for thesignal with a time interval to determine whether the time interval haspassed. The user interaction device continues to wait for reception ofthe wireless communications signal if the time interval has not passed.

If the result of act A206 is affirmative, the receiver proceeds to anact A208 to output a notification to the management device that thedevice failed to receive the wireless communications signal.

At an act A210, the receiver queries the management device of whether tooperate as a receiver, sender or waiter.

During the waiting for reception of the wireless communications signal,the receiver continually monitors for reception of the signal. If thewireless communications signal is detected at an act A212, the receiverproceeds to an act A214 to timestamp the time when the wirelesscommunications signal was received. The receiver returns to act A204 towait for the signal if the result of act A212 is negative.

At an act A216, the receiver communicates the timestamp of the receptionof the wireless communications signal to the management device. Thereceiver may also include its respective identifier which allows themanagement device to identify the receiver which received the signal.

At an act A218, the receiver receives instructions from the managementdevice which instructs the device to operate as a receiver, sender orwaiter during subsequent communications of additional wirelesscommunications signals.

At an act A220, the receiver enters the specified state of operation asa receiver, sender or waiter.

As mentioned above, the communicated wireless communications signalsused to determine refined location information may include differenttypes of signals (e.g., electromagnetic waves, sound waves). Theabove-recited methods of FIGS. 8 and 9 with respect to transmission andreception of signals may be implemented for the different types ofsignals. More specifically, the methods of FIGS. 8 and 9 may beperformed to implement communications of electromagnetic signals at onemoment in time and the methods may again be performed at another momentin time for communications of sound waves.

In one implementation, some of the wireless communications may beimplemented automatically without control of a user while others of thewireless communications may be implemented as a result of user control.In one more specific example, the user interaction devices may beconfigured to initiate and implement Bluetooth communicationsautomatically without user control while sound wave communications maybe initiated as a result of user control or instruction. The use ofsound waves may provide refined location information of improvedaccuracy in some embodiments compared with communications ofelectromagnetic waves since sound waves travel slower thanelectromagnetic waves.

The duration of the times of sound wave communications between thesending wireless communications device and the receiving wirelesscommunications devices may be multiplied by the velocity of sound (e.g.,340.29 m/s) to determine the distances of the respective receivingwireless communications devices with respect to the sending wirelesscommunications device. The speed of light may be used in calculationsfor the wireless communications of electromagnetic waves to determinethe distances of the receivers with respect to the sender. Thedetermined distances may be used as discussed below to provide refinedlocation information.

As mentioned above, some of the wireless communications devices may beWi-Fi communications devices which are arranged to wirelesslycommunicate with other wireless communications devices including theuser interaction devices. Typical Wi-Fi communications devices are notportable but are rather implemented in fixed static locations (e.g., hotspots). In one embodiment, the management device may store the locationsof Wi-Fi communications devices and may search for the presence of thesedevices which may be within a wireless communications range of one ofmore user interaction device which is implementing operations withrespect to augmented reality described herein.

The use of Wi-Fi communications devices which are in static locations isadvantageous since accurate information regarding these devices may bestored in the management device and used to accurately locate otherwireless communications devices (e.g., user interaction devices). Morespecifically, since the location information of these devices mayinclude reduced error compared with available location information ofportable devices, the location information of the static devices may beused to provide refined location information which may have increasedaccuracy with respect to the portable devices compared with operationsimplemented solely between portable devices to provide the refinedlocation information.

In one embodiment, portable user interaction devices may calculate theirrespective distances to static wireless communications devices which mayresult in reduced error in the determined locations of the portable userinteraction devices since the error of the locations of the staticwireless communications device in the physical world may be lesscompared with error present in available location information of theportable communications devices which would otherwise be used. Forexample, the location of one static device may be known with anincreased degree of accuracy and accordingly the processing ofinformation regarding communications with this device may providelocation information with an increased degree of accuracy compared withprocessing of communications with devices which may all be dynamicallymoving.

In one embodiment, the database of the management device may bepopulated with identification and location information regarding aplurality of Wi-Fi communications devices. In one embodiment, users ofan augmented reality community may upload location information to themanagement device of the Wi-Fi communications devices. For example,Wi-Fi communications devices may be programmed with their staticlocations upon installation in a facility and the information regardingthe static locations may be provided to the management device. Inanother example, user interaction devices having accurate informationregarding their present location may be positioned adjacent to the Wi-Ficommunications devices and the location of the user interaction devicemay be uploaded with the identification information of the Wi-Ficommunications devices to the management device. Any suitablearrangement may be used to provide the location information regardingthe Wi-Fi communications devices to the management device and which maybe stored therein for use in subsequent searching operations.

These Wi-Fi communications devices populated in the database of themanagement device may be subsequently searched at a later moment in timeusing initial location information of another user interaction device.The identified Wi-Fi communications devices which are proximatelylocated to one or more user interaction device may be used to implementcommunications with respect to the user interaction devices as discussedabove to provide refined location information of the user interactiondevices and which may have increased accuracy compared with locationinformation obtained by other methods, such as GPS.

In the example embodiment discussed above, the management devicedetermines the refined location information using information regardingthe wireless communications which are implemented between the wirelesscommunications devices. In another embodiment, the receiving wirelesscommunications devices (e.g., user interaction devices, Wi-Ficommunications devices) may communicate information regarding moments intime when the wireless communications devices received a wirelesscommunications signal which was emitted by a sender to the sender. Thesender may use the received information to determine its refinedlocation information using techniques described in further detail below(as opposed to having management device perform the calculations todetermine the refined location information).

As mentioned above, wireless communications signals including soundwaves may be utilized in attempts to improve the accuracy of the refinedlocation information of a plurality of wireless communications devices.The following is a discussion of processing of sound waves received by awireless communications device to identify wireless communicationssignals which were emitted by the sending wireless communications deviceaccording to one embodiment. As mentioned above, the receivers mayreceive a predefined frequency (i.e., the frequency is defined beforethe communication of the signal) from the management device and thereceivers may process received sound waves in attempts to determinewhether the predefined frequency is present in the received soundswaves.

The described processing may be performed upon output of a microphone ofa receiving user interaction device in one embodiment. The receivedsound may be converted into a function which can be tested to determineif the specified frequency of the communicated wireless communicationssignal is present in the received domain. Received sound is a sinusoidalwave and a Fourier Transform of Eqn. 1 may be used to convert thereceived sound into a desired domain in one embodiment:{circumflex over (f)}(φ)=∫_(−∞) ^(∞) f(x)e ^(−2πixφ) dx  EQN. 1Essentially f(x) is the received sound at time x and {circumflex over(f)}(φ) is the Fourier Transform for which φ is the frequency in hertzbeing checked against. If a strong frequency {circumflex over (φ)}exists within the ambient noise, then {circumflex over (f)}({circumflexover (φ)}) will present a discernable maxima within the graphed{circumflex over (f)}(φ).

{circumflex over (f)}(φ) uses a complex integration to determine theentire domain. A Riemann's Sum may be used to estimate this value asclosely to {circumflex over (f)}(φ) as possible. This is known as aDiscrete Fourier Transform where samples will be taken from the noise inthe room at a given interval. These samples may be reduced to:

$\begin{matrix}{{{- X_{k}} = {{\sum\limits_{n = 0}^{N - 1}{x_{n}{\mathbb{e}}^{{- \frac{2{\pi\mathbb{i}}}{N}}{kn}}\mspace{14mu} k}} = 0}},\ldots\mspace{14mu},{N - 1}} & {{EQN}.\mspace{14mu} 2}\end{matrix}$x_(n) is a sequence of complex numbers that go from n=0, . . . , n=N−1;i is the imaginary unit √{square root over (−1)}, and k is the frequencyto being checked against. For sound recognition, x_(n) would representthe amplitude of the ambient sound at a sample (time) n, and X_(k)represents the magnitude at frequency k.

In one example, received sound may be sampled at a rate of approximately44100 samples per second. The samples may be partitioned into differentgroups and the processing of Eqn. 2 may be performed on the groupsseparately. A resolution may be computed by dividing the sampling rateof 44100 samples per second by the number of samples obtained (e.g., anumber of obtained samples of 32768 provides a resolution of 1.34). Thequotient of the frequency of the wireless communications signal may bedivided by the resolution to provide the value of the result of thediscrete Fourier Transform if the sound wave is present in the soundreceived by the wireless communications device.

In one embodiment, a threshold for the magnitude of frequency may bespecified and the output of the processing for the frequency of interestmay be compared with this threshold. As an example, if the frequency ofinterest with the ambience is 20 Hz then Eqn. 2 becomes

$X_{20_{hz}} = {\sum\limits_{n = 0}^{N - 1}{x_{n}{\mathbb{e}}^{{- \frac{2{\pi\mathbb{i}}}{N}}20_{hz}n}}}$and X₂₀ _(hz) is compared with the threshold. If the threshold is met,then it can be assumed that the sound frequency of interest is beingcommunicated within the environment of the user interaction device. Inone embodiment, the operations of determining location using sound wavesmay be implemented at times specified by the user for example when thepresence of interfering sounds is low.

Once it is determined that the specified frequency of the wirelesscommunications signal has been received, further operations may beperformed in one embodiment to refine the processing of the time ofreception of the signal by the receiver. The group of samples in whichthe appropriate sound wave was detected may be split into halves and thetransform may be implemented on each half. The output of the DiscreteFourier Transform of the predefined frequency of the sound wave beinganalyzed should be smaller on the first half (if the frequency waspresent during the first and second halves) or only present in thesecond half (if the frequency was only present in the second half andnot the first half). The half indicating where the frequency wasreceived may be split into additional halves which may also be similarlyprocessed until the amount of error of the time when the frequency wasreceived is sufficiently small.

Referring to FIGS. 10 a-10 c, an example embodiment of a method forincreasing the accuracy of a user is shown. In one embodiment, a userinteraction device is configured to receive a GPS signal which providesthe device's Cartesian coordinates as well as an accuracy radius whichdefines an accuracy circle about the coordinates where the userinteraction device may exist. The example method described below reducesthe size of the accuracy circle (e.g., the area of possible locationsthe user interaction device may be with respect to the providedcoordinates) to provide refined location information regarding thelocation of the user interaction device with increased accuracy.

Referring to FIG. 10 a, a plurality of user interaction devices 100, 102are shown at different locations having respective coordinates as wellas respective accuracy circles 101, 103 based upon GPS signals receivedby the user interaction devices 100, 102. In addition, a line 106 isillustrated which corresponds to a linear distance between the userinteraction devices 100, 102 which was determined from processing ofwireless communications between user interaction devices 100, 102 asdescribed above. In one embodiment, the endpoints of the line 106 areprocessed with respect to the accuracy circles 101, 103 to reduce theareas of the accuracy circles 101, 103 where the user interactiondevices 100, 102 may be located and to provide the refined locationinformation regarding the locations of devices 100, 102 of increasedaccuracy compared with accuracy circles 101, 103.

In the described example, the user interaction devices 100, 102 existwithin their respective accuracy circles 101, 103 and the devices 100,102 are spaced a distance of the line 106 apart from one another. In oneembodiment, the method reduces the size of one of the accuracy circles101, 103 using the line 106 and the other of the accuracy circles 101,103.

Referring to FIG. 10 b, one example is described where a circle 107 isgenerated having a radius equal to the length of line 106. The accuracyof circle 103 may be reduced by identifying the union or overlap ofcircle 107 with circle 103 for all possible locations of device 100within circle 101.

More specifically, an end point of line 106 may be placed on the edge ofcircle 101 and the intersection or overlap of circles 103, 107 may bedetermined. Next, the intersections or overlap may be repeatedlydetermined for all locations of an end point of line 106 about the edgeof circle 101. After the intersections have been determined for circles103, 107 for all locations of the end point of line 106 on the edge ofcircle 101, a union of all the intersections may be determined whichindicates all possible locations of device 102 within accuracy circle103.

Referring to FIG. 10 c, the above-method may be repeated to identify allpossible locations of device 100 within circle 101 by using line 106 tocreate circle 108 and placing an endpoint of line 106 on differentlocations of the edge of circle 103 to identify all interactions ofcircles 101, 108 indicating all possible locations of device 100 withinaccuracy circle 101.

The above-described processing provides two new areas corresponding tothe intersections or overlaps of circles 101, 107 and 103, 108 and whichinclude all possible coordinate locations of respective devices 100,102. These two new areas may thereafter be used as updated accuracyareas for the devices 100, 102 for subsequent processing and whichreplace the initial accuracy circles 101, 103. These areas includesmaller numbers of possible locations of the user interaction devices100, 102 compared with the numbers of the possible locations of thedevices 100, 102 within accuracy circles 101, 103 providing refinedlocation information regarding the devices 100, 102.

In one embodiment, ellipsoids may be fitted to the new accuracy areasand the midpoints of the respective ellipsoids may be determined andused to indicate the new coordinates of the user interaction devices100, 102. The updated accuracy areas are smaller than the areas of theinitial accuracy circles 101, 103 providing refined location informationwhich may be used in combination with the newly determined respectivecoordinates of the user interaction devices 100, 102 from the midpointsof the ellipsoids. The original accuracy circles and coordinates of thedevices 100, 102 from the GPS signals may be disregarded and the newlydetermined accuracy areas and coordinate locations of the devices 100,102 may be stored within the management device and used for subsequentoperations, such as determining further refined location informationusing other user interaction devices, implementing augmented realityoperations, or other uses. Other embodiments and methods are possiblefor reducing the accuracy circles and providing new coordinates of thedevices 100, 102 in other embodiments.

The above-described example embodiment may also be implemented inarrangements where more than two user interaction devices are present.In one example method where three user interaction devices are present,the above-described processing may initially be implemented with respectto two of the devices to refine their accuracy circles/areas anddetermine new coordinates. After the processing with respect to thedevices, the above-described processing may be implemented with respectto one of the two devices (e.g., device 100) and an additional thirddevice (not shown in FIGS. 10 a-10 c). This processing was use therefined accuracy area and new coordinates of device 100 with theaccuracy area (e.g., circle) of the third device and a linear distancebetween the device 100 and the third device which may be determined byprocessing wireless communications between device 100 and the thirddevice. The new accuracy area and coordinates for device 100 and thethird device may be stored and thereafter used for subsequent processingoperations with respect to other devices.

The determined refined location information may be communicated to theuser interaction devices for use by the devices to accurately associateaugmented reality representations with respect to appropriate locationsof the physical world. As mentioned above, the augmented realityrepresentations may be associated with physical objects of the physicalworld. Additional details regarding usage of the refined locationinformation is discussed below. The refined location information mayalso be stored in a database of the management device along withtimestamp information which indicates a moment in time to which refinedlocation information pertains.

The above-discussion is an example embodiment of utilization of wirelesscommunications to implement operations with respect to determiningrefined location information which may have increased accuracy regardingthe locations of a plurality of user interaction devices compared withother available location information. As mentioned above, other methodsor techniques may also be utilized to also provide refined locationinformation regarding the locations of the user interaction devices inthe physical world. One method uses image recognition operations toprovide refined location information as discussed below in oneembodiment.

Referring to FIG. 11, one method is described which may be executed tocalculate refined location information regarding one or more userinteraction device using image recognition techniques according to oneembodiment. Processing circuitry of the management device may bearranged to implement the operations of the described example method. Inother arrangements, processing circuitry of the user interaction devicesmay be configured to implement one or more of the operations of thedescribed method (e.g., processing of captured images to locate markersor to compare images of a marker as discussed below). Furthermore, othermethods are possible including more, less and/or alternative acts.

At an act A300, the processing circuitry accesses initial locationinformation regarding a location of a user interaction device. Theinitial location information may be obtained using aGPS circuitry of theuser interaction device and which is communicated to the managementdevice in one embodiment.

At an act A302, the processing circuitry may search a database ofmarkers to locate one or more stored markers which may be viewable bythe user interaction device using the initial location information. Inother embodiments, initial location information of a user interactiondevice may be unavailable and the images captured by the userinteraction device may be communicated to the management device andanalyzed by the management device with respect to the stored images ofthe database. The use of initial location information improves the speedof the processing compared with some arrangements where initial locationinformation is not available or is otherwise not used.

At an act A304, the identified markers may be communicated from themanagement device to the user interaction device. The user interactiondevice may display one or more images of the markers to inform the userof the possible presence of the markers in their environment and theuser may capture images of the markers if the user find the markers inhis environment.

At an act A306, the management device accesses images obtained from theuser interaction device. In one example, the user of the userinteraction device may use the identified markers to capture one or moreimages of the markers which are observable by the users. The images maybe communicated to and accessed by the management device in act A306. Asmentioned above, in another embodiment, the user may capture a pluralityof images of their environment without knowledge of existing markers andthe management device may process the images in attempts to locate thepresence of the markers.

At an act A308, the processing circuitry performs image recognitionoperations to locate markers in the received images. In one embodiment,a database of known markers may be accessed and the processing circuitrymay search images captured by the user interaction device for thepresence of one or more of the known markers. The processing circuitrymay utilize location information regarding the user interaction deviceto identify known markers which are proximately located to the userinteraction device. The known markers may include images of physicalobjects in the physical world or symbols (e.g., geometric shapes such asa black outline of a bow tie on a white background). The images orsymbols may be pre-processed before use as markers and the pre-processedimages and symbols may be stored in the database for subsequentcomparison operations in some embodiments.

In one preprocessing example, the markers are pre-processed throughimage recognition (computer vision). First, key features of the markersare determined using a multi-scale Hessian detector algorithm which isreferenced in Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool,called “Speeded-Up Robust Features (SURF),” Computer Vision and ImageUnderstanding (CVIU), Vol. 110, No. 3, pp. 346-359, 2008, the teachingsof which are incorporated herein by reference. This processing providesdescriptors around keypoints which describe features of the markers forsubsequent comparison operations to indicate whether a match is present.In one embodiment, the keypoints and descriptors may be stored within adatabase of the management device.

Thereafter, the images received from the user interaction device mayalso be processed using the multi-scale Hessian detector and SURF fordetermining features through keypoints and descriptors of the images.The keypoints and descriptors resulting from the processing of theimages may be compared with the keypoints and descriptors of the knownmarkers which are stored in the database to determine whether the imagescontain one or more of the known markers.

In one comparison embodiment, the keypoints and descriptors of thefeatures of the images and the known markers are compared with oneanother to determine whether there is a sufficient match. A method fordetermining closely matched features is to compare the descriptors alongwith the edges of the keypoint to keypoint within the received images ofa scene or marker. One example method is referred to as FLANN and isdescribed in Marius Muja, and David Lowe, “Fast Approximate NearestNeighbors with Automatic Algorithm Configuration,” InternationalConference on Computer Vision Theory, 2009, the teachings of which areincorporated herein by reference. The comparison will locate knownmarkers present in the received images including location, pose, andscale of the markers.

At an act A310, the processing circuitry may process images captured bythe user interaction device and which include one or more markers withrespect to stored images of the markers to provide refined locationinformation regarding the location of the user interaction device in oneembodiment. The management device may include a database of images ofthe markers. The database may also include location information whichindicates various locations in the physical world from which the imagesof the markers in the database were captured from at previous moments intime. The images containing the markers captured by the user interactiondevice may be compared to the stored images of the markers which werecaptured from these known locations. The comparison of the imagesreceived from the user interaction device with the stored images and theusage of the known locations from which the stored images were capturedprovides refined location information regarding the locations of theuser interaction device when the images of the markers were captured bythe user interaction device. The refined location information may haveincreased accuracy regarding the locations of the user interactiondevice compared with the initial location information. In anotherexample, a size of the marker of an image captured from a known locationmay be compared with a size of the marker in an image captured by theuser interaction device and the results of the comparison may provideinformation regarding the location of the user interaction device.

In one embodiment using FLANN, the comparison of the images from theuser interaction device with the stored images of the markers determineswhether any of the images of the markers match. If one of the imagesfrom the user interaction device contains a marker which matches amarker in one of the stored images, then the refined locationinformation of the user is set to the location from which the matchingstored image was captured from.

In other examples, an image from the user interaction device may containa stored marker which may be detected by FLANN as being a match eventhough the image from the user interaction device may have been capturedfrom a different location than the image of the stored marker. Thefeatures of the stored marker may be modified using a perspectivetransform to match the features of the marker within the image from theuser interaction device. The perspective transform modifies the featuresof the stored marker to appear as if the image was captured fromdifferent locations. Once the modifications by the perspective transformidentify a match of the modified features of the stored marker with thefeatures of the marker in the image from the user interaction device,the offset of the user's location from which the user captured the imageis known with respect to the location from which the image of the storedmarker was obtained and may be used to provide refined locationinformation of the user interaction device. More specifically, in thisexample, the refined location information is set to the location fromwhich the image of the stored marker was captured as modified by theoffset which was determined by the perspective transform processing.

At an act A312, the images received from the user interaction devicesand which include the markers (and the determined location informationof the user interaction device when the images were captured) may bestored in the database and used for future image processing operationsof subsequently captured images to provide location information ofdevices which captured the subsequent images.

In one embodiment, it is desired to have accurate information regardingthe locations from which the images were captured and informationregarding the accuracy of the location information may be used todetermine whether images are stored in the database and used forsubsequent image processing operations to provide location informationof other users. Furthermore, accuracy resulting from the processing ofimages may be improved when a plurality of markers are present in theimages for processing or a plurality of different images of a singlemarker which were obtained from different locations are available forprocessing. In some embodiments, a user may upload a plurality of imagesof a marker which may be slightly rotated from one another or observedfrom slightly different locations.

At an act A314, the refined location information may be communicated toone or more of the user interaction devices and used by the devices forimplementing operations with respect to experiencing augmented realityor other uses. The refined location information may also be storedwithin the database which includes the wireless communications devicesand their respective locations.

As mentioned above, other methods are possible and some of the acts maybe performed by different entities or omitted. In another example, themarkers may not be communicated to the user interaction device, butrather the user interaction device may capture images of its environmentand the images may be uploaded to the management device for processingto locate markers and provide refined location information using thelocated markers by comparison with existing images in a database of themanagement device.

In another embodiment, images of markers from the database of themanagement device may be communicated to the user interaction devicesand used by the devices to calculate refined location information. Inone embodiment, the user interaction device may be configured to processimages captured by the user interaction device including performingobject recognition operations to identify the presence of markers and tocompare the captured images with the images from the database to providethe refined location information as discussed above with respect tooperations of the management device.

The management device may have greater processing capacity than theprocessing capacities of the user interaction devices and implementingprocessing of images using the management device may be faster thanprocessing of the images using the user interaction devices.

The above discussion describes various methods for determining refinedlocation information regarding locations of user interaction devices inthe physical world. The refined location information may be used toaccurately associate augmented reality representations at appropriatelocations of the physical world as captured by the user interactiondevices. For example, the refined location information of the pluralityof user interaction devices may be used to associate augmented realityrepresentations with physical objects of the physical world.

In one example, a plurality of markers may be associated with aplurality of respective virtual objects. Accordingly, when a marker islocated in an image of a scene, the user interaction device may depictthe virtual object in the image by replacing the marker 14 with thevirtual object representation in images depicted using the display(e.g., the puppy representation 18 replaces the marker 14 shown in FIG.1).

Improvements in processing capabilities and communications bandwidthhave led to new implementations and applications of augmented realitysystems. Provision of accurate location information utilizing apparatusand methods described herein according to example embodiments of thedisclosure enable computing systems, including portable devices, toimplement improved augmented reality operations. At least some aspectsof the disclosure provide systems, apparatus and methods which enablelocations of user interaction devices to be determined with increasedaccuracy permitting augmented reality representations to be referencedwith respect to the physical world with increased accuracy compared withusage of location information determined by conventional methods such asGPS which has inherent limited accuracy and has yet larger error in someareas (e.g., indoors). Accurate location information of the augmentedreality user interaction devices enables augmented realityrepresentations to be correctly referenced with respect to the physicalworld such that different user interaction devices experiencing theaugmented reality representations will portray the augmented realityrepresentations correctly with respect to the physical world (i.e., atappropriate locations with respect to physical world locations andobjects).

Some conventional location technologies including GPS have shortcomingswith respect to use in augmented reality systems as discussed above.More specifically, civilian entities are currently precluded fromobtaining the highest accuracy possible from GPS. In addition, theaccuracy of GPS is further degraded at indoor locations or at otherlocations where the signals from the satellites of the GPS system mayencounter obstacles or may not be clearly received. Cell towertriangulation has been used in conventional arrangements to augment GPSreadings but this methodology also has shortcomings of being of limiteduse indoors or in areas which are lacking a sufficient number of celltowers.

At least some aspects of the disclosure include obtaining and combininglocation information from different techniques in attempts to accuratelylocate the positions of the user interaction devices. Locationinformation obtained from aspects of the disclosure may be usedseparately or in combination with conventional location determinationarrangements in some embodiments.

The protection sought is not to be limited to the disclosed embodiments,which are given by way of example only, but instead is to be limitedonly by the scope of the appended claims.

Further, aspects herein have been presented for guidance in constructionand/or operation of illustrative embodiments of the disclosure.Applicant(s) hereof consider these described illustrative embodiments toalso include, disclose and describe further inventive aspects inaddition to those explicitly disclosed. For example, the additionalinventive aspects may include less, more and/or alternative featuresthan those described in the illustrative embodiments. In more specificexamples, Applicants consider the disclosure to include, disclose anddescribe methods which include less, more and/or alternative steps thanthose methods explicitly disclosed as well as apparatus which includesless, more and/or alternative structure than the explicitly disclosedstructure.

The invention claimed is:
 1. A location determination method comprising:accessing first location information regarding a location of a userinteraction device in a physical world, wherein the user interactiondevice is configured to generate an augmented reality representationwith respect to the physical world; using the first locationinformation, identifying a marker which is proximately located withrespect to the location of the user interaction device; accessing animage generated by the user interaction device which includes themarker; and processing the image to determine second locationinformation regarding the location of the user interaction device, andwherein the second location information has increased accuracy withrespect to the location of the user interaction device in the physicalworld compared with the first location information.
 2. The method ofclaim 1 wherein the processing comprises processing the image generatedby the user interaction device with respect to an image of the marker.3. The method of claim 2 wherein the processing comprises processing theimage generated by the user interaction device using informationregarding a known location from which the image of the marker wascaptured.
 4. The method of claim 1 wherein the accessing the firstlocation information and the identifying are implemented using amanagement device which is in wireless communication with the userinteraction device.
 5. The method of claim 4 further comprising storingan image of a marker received from the user interaction device in adatabase.
 6. The method of claim 4 wherein the identifying comprisessearching a database which includes the marker using the first locationinformation.
 7. The method of claim 4 further comprising communicatingthe marker from the management device to the user interaction device,and wherein the processing comprises processing using the userinteraction device.
 8. The method of claim 4 further comprising:communicating the marker from the management device to the userinteraction device; and displaying the marker using the user interactiondevice.
 9. The method of claim 4 wherein the accessing the image and theprocessing are implemented using the management device.
 10. The methodof claim 9 further comprising communicating the second locationinformation to the user interaction device.
 11. A computing systemcomprising: communications circuitry configured to implementcommunications externally of the computing system; and processingcircuitry coupled with the communications circuitry, and wherein theprocessing circuitry is configured to access first location informationreceived by the communications circuitry regarding a location of a userinteraction device in a physical world, to identify a marker which isproximately located with respect to the location of the user interactiondevice, to access a plurality of images generated by the userinteraction device, and to process the images with respect to the markerto determine second location information regarding the location of theuser interaction device and which has increased accuracy with respect tothe location of the user interaction device in the physical worldcompared with the first location information.
 12. The system of claim 11wherein the processing circuitry is configured to process the imagesgenerated by the user interaction device with respect to an image of themarker which was captured from a known location to determine the secondlocation information.
 13. The system of claim 12 wherein the processingcircuitry is configured to process the images generated by the userinteraction device using information regarding the known location fromwhich the image of the marker was captured.
 14. The system of claim 11wherein the processing circuitry is configured to process the imagesgenerated by the user interaction device to locate the marker in atleast one of the images.
 15. The system of claim 11 further comprisingstorage circuitry comprising a database of markers, and wherein theprocessing circuitry is configured to search the database using thefirst location information to identify the marker.
 16. The system ofclaim 11 wherein the marker comprising an initial marker, and furthercomprising storage circuitry comprising a database including the initialmarker and a plurality of additional markers, and wherein thecommunications circuitry is configured to receive at least one of theadditional markers from externally of the computing system.
 17. Thesystem of claim 11 wherein the processing circuitry is configured tocontrol the communications circuitry to communicate the marker to theuser interaction device.
 18. The system of claim 11 further comprisingstorage circuitry configured to store augmented data comprising anaugmented reality representation, and wherein the processing circuitryis configured to control the communications circuitry to communicate theaugmented data to the user interaction device.
 19. The system of claim18 wherein the augmented reality representation comprises a virtualobject.
 20. An article of manufacture comprising: non-transitory mediacomprising programming which causes processing circuitry to performprocessing comprising: accessing first location information regarding alocation of a user interaction device in a physical world; identifying amarker which is proximately located with respect to the location of theuser interaction device; accessing a plurality of images generated bythe user interaction device; and processing the images with respect tothe marker to determine second location information regarding thelocation of the user interaction device and which has increased accuracywith respect to the location of the user interaction device in thephysical world compared with the first location information.